Papers with model interpretability
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
Text Classification through Glyph-aware Disentangled Character Embedding and Semantic Sub-character Augmentation (2020.aacl-srw)
Copied to clipboard
| Challenge: | a character-based text classification framework for non-alphabetic languages is proposed . glyphs are used to give visual meaning to characters, such as "" and "." |
| Approach: | They propose a character-based text classification framework for non-alphabetic languages . they propose glyph-aware disentangled character embeddings that are dimensionally independent . |
| Outcome: | The proposed framework improves the performance of Japanese text classification tasks . it provides embedding interpretability and performance competitively with the state-of-the-art model . |
Improving Knowledge Graph Embedding Using Simple Constraints (P18-1)
Copied to clipboard
| Challenge: | Recent efforts focused on designing more complicated models or incorporating extra information beyond triples. |
| Approach: | They propose to use non-negativity constraints on entity representations and approximate entailment constraints on relation representations to improve KG embedding. |
| Outcome: | The proposed model outperforms baseline models on WordNet, Freebase, and DBpedia. |
Is Sparse Attention more Interpretable? (2021.acl-short)
Copied to clipboard
| Challenge: | Sparse attention has been claimed to increase model interpretability . however, the attention distribution is typically over representations internal to the model rather than the inputs themselves . |
| Approach: | They conduct experiments to understand how sparsity affects our ability to use attention as an explainability tool. |
| Outcome: | The proposed model does not map to a sparse set of influential inputs, but rather to fewer inputs. |
AbductionRules: Training Transformers to Explain Unexpected Inputs (2022.findings-acl)
Copied to clipboard
| Challenge: | AbductionRules is a set of natural language datasets designed to train and test generalisable abduction over natural-language knowledge bases. |
| Approach: | They propose to train and test generalisable abduction over natural-language knowledge bases by using natural language datasets to fine tune pre-trained Transformers. |
| Outcome: | The proposed models learned generalisable abduction techniques but also exploited the structure of the datasets. |
Dialz: A Python Toolkit for Steering Vectors (2025.acl-demo)
Copied to clipboard
| Challenge: | *Dialz* is a Python library for advancing research on steering vectors for open-source LMs. |
| Approach: | They propose a Python library for advancing research on steering vectors for open-source LMs. |
| Outcome: | The proposed method reduces harmful outputs and provides insights into model behaviour across different layers. |
Locally Distributed Activation Vectors for Guided Feature Attribution (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to explain predictions of deep neural networks are unstable and do not always provide faithful explanations to the target model. |
| Approach: | They propose a method to learn explanations-specific representations while constructing deep network models for text classification. |
| Outcome: | The proposed method improves model interpretability while preserving predictive performance. |
Understanding Game-Playing Agents with Natural Language Annotations (2022.acl-short)
Copied to clipboard
| Challenge: | a new dataset contains 10K human-annotated Go games. |
| Approach: | They propose to use natural language annotations to predict game terms from Go models . they use linear probing to predict domain-specific terms from the model's intermediate state representations . |
| Outcome: | The proposed model can predict domain-specific terms from human-annotated Go games . the proposed model is based on a dataset containing 10K human-rannotated games based in the game . |
SAFR: Neuron Redistribution for Interpretability (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on controlling neuron distribution for interpretability have focused on focusing on monosemanticity instead of focusing solely on feature interactions. |
| Approach: | They propose a method to regularize feature superposition by encoding representations of multiple features within a single neuron. |
| Outcome: | The proposed method improves model interpretability without compromising prediction performance. |
Prompting for explanations improves Adversarial NLI. Is this true? {Yes} it is {true} because {it weakens superficial cues} (2023.findings-eacl)
Copied to clipboard
| Challenge: | Explanation prompts are used to generate an explanation for a given input . they are also used to improve model performance on adversarial benchmarks . |
| Approach: | They propose to use explanation prompts to generate an explanation that supports a label . they argue that prompting for explanations weakens superficial cues . |
| Outcome: | The proposed explanation prompts outperform label-only prompts on adversarial benchmarks. |
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use natural language rationales provide intuitive, higher-level explanations that are easily understandable by humans. |
| Approach: | They propose a model that generates free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. |
| Outcome: | The proposed model generates free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. |
On the Feasibility of In-Context Probing for Data Attribution (2025.findings-naacl)
Copied to clipboard
| Challenge: | In-context probing (ICP) can be used to identify training data that contributes to model outputs, but many data attribution methods, such as influence functions, use model gradients and are computationally expensive. |
| Approach: | They propose to use in-context probing (ICP) to proxy for gradient-based data attribution for data selection under conditions contingent on data similarity. |
| Outcome: | The proposed method can be used to identify training data that contribute to model outputs and fine tune models on training data. |
Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for predicting research replication are insufficient especially for long research papers. |
| Approach: | They propose to build an interpretable neural model which can provide sentence-level explanations and apply weakly supervised approach to leverage large corpus of unlabeled datasets. |
| Outcome: | The proposed model can provide sentence-level explanations and leverage large unlabeled datasets to boost interpretability and improve prediction performance. |
The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models have proven effective for machine translation, but at the expense of interpretability. |
| Approach: | They analyze how morphological features are captured at different levels of the NMT encoder while varying the target language. |
| Outcome: | The proposed model is not interpretable, but only captures morphological features in context and only to the extent they are directly transferable to the target words. |
On the Lack of Robust Interpretability of Neural Text Classifiers (2021.findings-acl)
Copied to clipboard
Muhammad Bilal Zafar, Michele Donini, Dylan Slack, Cedric Archambeau, Sanjiv Das, Krishnaram Kenthapadi
| Challenge: | Several models have been proposed to interpret models with feature-based interpretability methods. |
| Approach: | They propose to quantify the robustness of neural text classifiers by using two randomization tests to compare models with identical initializations. |
| Outcome: | The proposed methods show surprising deviations from expected behavior . the results raise questions about the extent of insights that practitioners may draw from interpretations. |
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving model interpretability require prior information or human annotations as additional inputs. |
| Approach: | They propose a variational word mask method to automatically learn task-specific important words and reduce irrelevant information on classification, which ultimately improves model interpretability. |
| Outcome: | The proposed method improves model prediction accuracy and interpretability on seven datasets. |
MedThink: A Rationale-Guided Framework for Explaining Medical Visual Question Answering (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models for medical visual question answering are limited in their interpretation and interpretation . a semi-automated annotation process is used to streamline data preparation and build new benchmark datasets . |
| Approach: | They propose a semi-automated annotation process to streamline data preparation and build new benchmark Med-VQA datasets. |
| Outcome: | The proposed method achieves an accuracy of 83.5% on R-RAD, 86.3% on RSLAKE and 87.2% on RPath. |
Investigating the Benefits of Free-Form Rationales (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourced rationales provide additional background knowledge to models . a qualitative study shows generated rationale is not as useful for humans as crowdsourced ones . |
| Approach: | They investigate whether crowdsourced rationales provide additional background knowledge to models . they find that ECQA rationale provides additional background information to understand a decision . |
| Outcome: | The results show that ECQA rationales provide additional background knowledge to understand a decision . compared to crowdsourced rationale, generated rationale is not as useful for humans . |
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? (2020.acl-main)
Copied to clipboard
| Challenge: | a new study examines the impact of algorithmic explanations on simulatability of machine learning models . a model is simulatable when a person can predict its behavior on new inputs . |
| Approach: | They conduct human subject tests to isolate effect of algorithmic explanations on simulatability . they find ratings of explanations are not predictive of how helpful they are . |
| Outcome: | The results provide the first reliable estimates of how explanations influence simulatability . they show that ratings are not predictive of how helpful explanations are . |
Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability (2025.coling-main)
Copied to clipboard
| Challenge: | a new study explores how large language models capture aspects of human linguis-tic ability . large language model performance is limited by the mechanisms behind their performance . |
| Approach: | They employ psycholinguistic paradigms to explore neuron-level representations in language models . they found that large language models exhibit human-like abilities in three tasks . |
| Outcome: | The proposed model shows human-like abilities in sound-shape association, gender association and implicit causality. |
A Collaborative Reasoning Framework Powered by Reinforcement Learning and Large Language Models for Complex Questions Answering over Knowledge Graph (2025.coling-main)
Copied to clipboard
| Challenge: | Knowledge Graph Question Answering (KGQA) aims to answer natural language questions by reasoning across multiple triples in knowledge graphs. |
| Approach: | They propose a collaborative reasoning framework powered by RL and LLMs to answer complex questions based on the knowledge graph. |
| Outcome: | The proposed model surpasses state-of-the-art models on four datasets. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are often judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. |
| Approach: | They propose a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) AIS quantifies how much each ability contributes to a model’s success on a given benchmark. |
| Outcome: | The proposed framework decomposes performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model’s success on a given benchmark. |
Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Contemporary advances in NLP are built on the representational power of latent embedding spaces learned by self-supervised language models (LMs). |
| Approach: | They use a new information theoretic probing suite to analyze representational subspaces in language models. |
| Outcome: | The proposed approach compared performance of nine tasks across 2M pre-training steps and five seeds. |
PsyPath: Psychologically-guided Self-Exploration for Personality Detection (2026.findings-acl)
Copied to clipboard
| Challenge: | Personality detection aims to label traits via identifying linguistic cues from written text. |
| Approach: | They propose a framework that allows large language models to generate and answer psychologically meaningful questions and a hybrid scoring mechanism to evaluate the generated nodes in the reasoning paths. |
| Outcome: | The proposed framework outperforms baselines on two benchmark datasets and significantly improves performance and interpretability in downstream tasks. |
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities. |
| Approach: | They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. |
| Outcome: | The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs. |
A Causal Lens for Evaluating Faithfulness Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability, but they may not reflect the model’s truereasoning faithfully. |
| Approach: | They propose a testbed framework for evaluating faithfulness metrics for natural language explanations using diagnosticity and model-editing methods. |
| Outcome: | The proposed framework evaluates faithfulness metrics for natural language explanations on four tasks including fact-checking, analogy, object counting, and multi-hop reasoning. |